Interactively pausing the broadcast stream displayed, graphical generation of telestrator data queries designates the location of the object in the portion of the transmitted still image frame

ABSTRACT

Methods, apparatuses, and systems enabling an interactive information retrieval system that includes a user interface allowing users to graphically generate information queries for objects of interest found in various media, such as television, video, and print. The invention can allow users to designate objects of interest within a television or video display and receive information related to the objects of interest. The invention can also allow users to generate queries designating objects of interest within a graphical image using a GUI.

CROSS-REFERENCE TO RELATED APPLICATION

The present U.S. application Ser. No. 11/618,272 filed Dec. 29, 2006 isa continuation of U.S. application Ser. No. 10/670,998 filed Sep. 24,2003, is now U.S. Pat. No. 7,240,075 claims Priority from ProvisionalApplication Ser. No. 60/413,113 filed Sep. 24,2002.

FIELD OF THE INVENTION

The present invention relates to interactive television or computersystems, namely, methods, apparatuses, and systems enabling aninteractive information retrieval system featuring a graphicaluser-interface (GUI) for selecting objects of interest shown within animage, such as a television or other video image frame, print media, andthe like.

BACKGROUND OF THE INVENTION

There are myriad documents and files accessible over the Internet.Retrieving desired information on the Internet, however, generallyrequires knowledge of an associated Uniform Resource Locator (URL) ordata location. For example, if a consumer wishes to obtain informationabout or order a particular company's product on the World Wide Web(WWW), she must know the URL corresponding to that company's web site.Conversely, if a corporation wants the public to visit a web sitecontaining information about its products, the corporation willtypically advertise its web site and corresponding URL in television,radio, print or other media. A consumer may then enter this URL,assuming he remembers it, into a web browser and access the web site.

When a specific URL or data location is not known, search engines are away of locating desired information. A user typically enters key wordsor search terms into a search engine, which returns a list of URLscorresponding to web sites or USENET groups where the key words orsearch terms were found. A search engine will often return a large listof web sites, through which the user must wade to locate the few websites relevant to her query.

Due in part to the proliferation of commercial web sites, consumers havebecome accustomed to the notion that there is a web site containinginformation for the vast majority of products and services beingcommercially offered. Yet, as described above, access to a particularweb site on the Internet requires knowledge of the actual URL or accessto a search engine. This becomes problematic when there is no immediateaccess to a computer connected to the Internet. For example, when aperson sees an item displayed on a television and desires furtherinformation, he must remember characteristics about the item. At a latertime, the person can enter those characteristics as search terms in atypical search engine. Alternatively, assuming the broadcastadvertisement or program displays an ‘800’ number, the user may use thenumber and access a telephone operator or call center to obtain moreinformation and/or order the item. Beyond these methods, however, thereare few ways of efficiently identifying data locations or URLs, oraccessing other relevant resources, based upon an observation of aparticular object or event. In light of the foregoing, it can be seenthat a need exists for alternative methods of identifying URLs, or otherdata locations on a computer network, offering information correspondingto objects displayed in other media, such as television broadcasts andprint.

SUMMARY OF THE INVENTION

The present invention provides methods, apparatuses, and systemsenabling an interactive information retrieval system that includes auser interface allowing users to graphically generate informationqueries for objects of interest found in various media, such astelevision, video, and print. The invention can allow users to designateobjects of interest within a television or video display and receiveinformation related to the objects of interest. The invention can alsoallow users to generate queries designating objects of interest within agraphical image using a GUI.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram setting forth an interactiveinformation retrieval system according to an implementation of theinvention.

FIG. 2 depicts a client device including a user interface for use inconnection with an implementation of the invention.

FIG. 3A is a flow chart illustrating a method, according to animplementation of the invention, directed to interacting with a receiverto designate a selected region within an image frame.

FIG. 3B is a flow chart illustrating a method, according to animplementation of the invention, directed to processing telestratorinput from a client device and transmitting information lookup requests.

FIG. 4 is an object locator table including positional parameterscorresponding to a selected region in relation to frame identifiers orframe set identifiers.

FIG. 5 is an object information table storing computer network addressesin association with object identifiers for a plurality of content types(e.g., information, ads, and coupons).

FIG. 6 is a flow chart illustrating a method, according to animplementation of the invention, for identifying objects in response toa lookup request and retrieving information related to the objects.

FIG. 7 is a functional block diagram setting forth an interactiveinformation retrieval system according to an implementation of theinvention.

DESCRIPTION OF PREFERRED EMBODIMENT(S)

FIG. 1 illustrates an interactive information retrieval system 10according to one implementation of the invention. The informationretrieval system operates in connection with a computer network 20. Inthe implementation shown in FIG. 1, computer network 20 comprises anInternet Protocol (IP) network, such as the Internet. Computer network20 can be any suitable computer network, including but not limited to anopen, wide-area network (WAN), an electronic network, an opticalnetwork, a wireless network, and/or any combination thereof. Computernetwork 20 can be a packet-based communications environment, employingTCP/IP protocols and having a plurality of interconnected digital packettransmission stations operative to route data between TCP/IP endsystems. The invention, however, has applications in computer networkenvironments employing any suitable transport layer and network layerprotocols.

As FIG. 1 illustrates, implementations of the invention can operate in adistributed computing environment. In one implementation, thedistributed computing environment can include a direct broadcastsatellite (DBS) system 30 that comprises a satellite broadcast center31, a satellite 32, a satellite dish 34, a satellite receiver 36A, adisplay device such as a television 38, and a client device 90. Inanother implementation, the distributed computing environment caninclude a cable television system 40 that comprises a cable televisionhead end system 42, a cable receiver 36B, a display device such as atelevision 38, and client device 90. And in further implementations, thedistributed computing environment can include a client computer 45 thatis coupled to the computer network 20 through a cable modem 48 or aDigital Subscriber Line (DSL) modem (not shown). The distributedcomputing environment also includes an information lookup system 50.

An implementation of client device 90 includes a display, a userinterface, and functionality for wireless communications. Thisfunctionality can be implemented using, for example, infrared signals(e.g., IrDA), radio signals (e.g., 802.11 technologies (Wi-Fi) orBluetooth), short burst Ethernet, and message services such as ShortMessage Service (SMS) and Multimedia Message Service (MMS). Inimplementations of the invention, client device 90 can comprise apersonal digital assistant (PDA), a mobile phone, a combination of a PDAand a mobile phone, or other wireless devices. In some implementations,client device 90 can include a digital camera.

Information LookUp System

Information lookup system 50 can interact with client device 90, asdescribed more fully below, to retrieve information related to an objectof interest designated by a user using client device 90. In oneimplementation, information lookup system 50 comprises at least oneapplication server 52, a user account database 54, and an informationdatabase 56.

Application server 52 includes functionality implementing the processflows described herein. User account database 54 stores informationrelated to at least one user account. In one implementation, such useraccount information includes, but is not limited to, a user accountidentification, name, address, e-mail address, fax number (and,optionally, other personal contact information), as well as systemsettings, preferences, psychographic information, demographicinformation, and behavioral information obtained through monitoring auser's use of the system described herein. For example, user accountdatabase 54 can store information related to the user's age, maritalstatus, children, cars owned, income level, home ownership status (e.g.,owner vs renter), time at current residence, etc. User account database54 can also store information gleaned from the use of the informationretrieval system by the user as described herein, such as a log ofinformation requests, etc. User account database 54 can further scorefinancial account infor information for at least one user to facilitatefinancial transactions.

Information database 56 stores information related to one or moreobjects embodied in images (i.e., still or video images) that arebroadcast, streamed, transmitted, printed, or otherwise disseminated tousers and displayed on a display device. such as a television 38 or themonitor associated with client computer 45, or in print media.

Direct Broadcast Satellite System

DBS system 30 is operative to provide broadcast satellite televisionservices to subscribers. Satellite broadcast center 31 receives incomingtelevision programming from companies (e.g., CNN and ESPN) and thenconverts the television programming into an encoded digital television(DTV) broadcast signal. The encoding format used is generally MPEG-2. Toreach all of its subscribers, DBS system 30 employs one one or moresatellites 32 that are in geosynchronous orbits that Earth to deliverthe broadcast signal. The encoded DTV broadcast signal is sent fromsatellite broadcast center 31 to one or more of these orbitingsatellites 32 and the broadcast signal is then rebroadcast back down tosubscribers on Earth who can receive the broadcast signal usingsatellite dish 34. The broadcast signal is captured by satellite dish 34and then transmitted to satellite receiver 36A where the broadcastsignal is decoded and displayed on television 38. The decoded broadcastis made up of a number of image frames that are displayed in rapidsuccession.

In one implementation, satellite receiver 36A comprises one or moremicroprocessors for running an operating system (such as Linux orWindows CE), and for parsing the video data transport stream (e.g., MPEGstreams, etc.) within the broadcast signal. Satellite receiver 36A canalso include RAM, an MPEG decoder chip, and other chips for audiodecoding and processing. However, as one skilled in the art willrecognize, the exact contents of a satellite receiver depend on the DTVstandard used. Satellite receiver 36A can further include a hard drivefor storing recorded television broadcasts, downloaded software, andother applications in an implementation, satellite receiver 36A can bephysically integrated into television 38 or any other display device.

In one implementation, satellite receiver 36A can transmit data in theupstream direction through a telephone line back to another portion ofDBS system 30, such as satellite broadcast center 31. This completes acommunication loop between satellite receiver 36A and satellitebroadcast center 31 (i.e., satellite receiver 36A can receive broadcastsignals from DBS system 30 through satellite dish 34 and can alsotransmit signals back to DBS system 30). Satellite receiver 36A can alsoinclude a telestrator application that is invoked when a command signalis received from client device 90, as discussed more fully below.

Cable Television Data Transmission System

Cable television system 40 is operative to provide cable televisionservices and cable modem services to subscribers. In one implementation,cable head end system 47 can include a satellite dish antenna (notshown) for receiving incoming programming, which is passed on to thesubscriber. In one embodiment, cable head-end system 42 further includesinternet access functionality, including a Cable Modem TerminationSystem (CMTS) (which sends and receives digital cable modem signals on acable network) to provide Internet services to subscribers.

Cable modem 48 communicates with the CMTS associated with cable head-endsystems 42 to provide client computer 45 with access to resources oncomputer network 20. Cable modem 48 can be an external device (as shown)or it can be integrated within client computer 45.

Cable television system 40 includes a cable receiver 36B that receivesand decodes DTV broadcasts transmitted from cable head-end system 42.Like the satellite broadcast, the decoded cable broadcast is made up ofa number of image frames that are displayed in rapid succession. In oneembodiment, similar to satellite receiver 36A, cable receiver 36Bcomprises one or more microprocessors for running an operating system(such as Linux or Windows CE), and for parsing the video data transportstream (e.g., MPEG streams, etc.). Cable receiver 36B can also includeRAM, an MPEG decoder chip, other chips for audio decoding andprocessing, and a hard drive for storing recorded television broadcasts,downloaded software, and other applications. In an implementation, cablereceiver 36B can be physically integrated into television 38 or anyother display device.

Again, the exact contents of a cable receiver depend on the DTV standardused. In one implementation, cable receiver 36B transmits data in theupstream direction through a telephone line, rather than the cableconnection. And similar to satellite receiver 36A, in one implementationcable receiver 36B can include a telestrator application that is invokedwhen a command signal is received from personal digital assistant 90, asdiscussed more fully below.

Satellite receiver 36A and cable receiver 36B (collectively referred toherein as “receiver 36”) can each include functionality for wirelesslycommunicating with client device 90. In one implementation, clientsoftware can enable client device 90 to transmit “pause” commands, andreceiver 36 can include functionality to pause a satellite broadcast, orcable broadcast when such pause commands are received. When a broadcastis paused in this manner, an image frame is displayed that correspondsto the moment when the pause command was received.

Receiver 36 is also operative to receive and process signals from clientdevice 90 that designate a selected region of the image frame. Inimplementations of the invention, receiver 36 can display the selectedregion of the image frame or overlay the image frame with markings orsymbols that designate the selected region. As discussed more fullybelow, receiver 36 can further include functionality for communicatingwith information lookup system 50 to provide information, if any,associated with one or more objects contained within the selected regionof the image frame designated by the user. In one implementation,receiver 36 can include a real-time clock and functionality thatcommunicates with either DBS system 30 or cable television system 40 tosynchronize the real-time clock with a central clock. The central clockcan be maintained at a location such as DBS system 30 or cabletelevision system 40.

To facilitate initiation and maintenance of user accounts, users canmanage their respective user accounts by accessing lookup system 50through a network access device (not shown) connected to computernetwork 20. In one implementation, the network access device is abrowser executed on client computer 45 or a network computer (notshown). In one implementation, page-based interfaces for accessinginformation lookup system 50 are transmitted to client computer 45having a browser and a connection to computer network 20.

Client computer 45 can comprise any computer, special-purpose computingdevice, or any other suitable device having the required functionality.In one implementation, client computer 45 includes at least oneprocessor, a data storage system (including volatile and non-volatilememory), at least one input device (e.g., a keyboard), and at least oneoutput device (e.g., a display). In one implementation, client computer45 is connected to computer network 20 through a modem connection (e.g.,dial-up, DSL, or cable) or through a network line. Such connections canbe wireless as well. In addition, although implementations of the systemare described as working in conjunciton with a browser, any suitabledevice, application, or client software for receiving, displaying, andtransmitting data over a computer network can be used. In oneimplementation, the browser implemented on client computer 45 supportsthe SSI. (‘Secure Sockets Layer’) protocol, the S-HTTP (‘Secure HTTP’)protocol, or any other similar protocol for transmitting confidential orprivate information over an open computer network. Users are individualsor other legal entities having the capacity to possess financialaccounts, such as corporations, partnerships, non-profit organizations,trusts, and the like.

Client Device

FIG. 2 illustrates one implementation of client device 90 as a personaldigital assistant (PDA), which is a mobile hand-held device thatprovides computing and information storage and retrieval capabilitiesusing applications installed therein. Client device 90 can be apen-based device, using a stylus 92 and a touch sensitive screen 94(rather than a keyboard) for input. Client device 90 further includes aninfrared port for wireless communication of data with a second device,such as receiver 36. In other implementations, client device 90 caninclude speech recognition functionality allowing for a voice-based userinterface operative to receive voice commands (e.g., “pause” command,see below) from users, or can use other methods of wirelesscommunication, such as Bluetooth or 802.11 technologies.

FIG. 2 also illustrates one implementation of a user interface 96 thatcomprises an installed application (i.e., client software) enabling auser to transmit commands to receiver 36 that cause the receiver toperform actions such as pausing a broadcast and designating selectedregions on the image frame displayed on television 38. Oneimplementation of user interface 96 can include a screen display box102, a “pause” button 104, a “done” button 106, and a “cancel” button108.

Screen display box 102 can have dimensions that correspond to the aspectratio of a standard television display (e.g., 4×3 or 16×9); however, anysuitable aspect ratio can be used depending on the display device type.As discussed below, screen display box 102 provides an area in which auser can draw or designate a selected region 120 within an image framedisplayed on a display device, such as television 38. This selectedregion 120 contains one or more object of interest for which the userdesires further information.

Activation of “pause” button 104 causes client device 90 to transmit apause command via a wireless signal, and activation of “done” button 106causes client device 90 to transmit data defining selected region 120via wireless signals. These wireless signals can be received andprocessed by receiver 36. “Cancel” button 108 allows the user to cancelprevious commands and allows receiver 36 to resume normal broadcastdisplay operations.

Operation of Client Device and Receiver

In one implementation of the invention, a user can use client device 90to pause a live video broadcast (e.g., a cable broadcast or a satellitebroadcast), designate one or more selected regions within an image frameshown on a display device, and transmit a query through receiver 36 toinformation lookup system 50 to retrieve information, if any,corresponding to the query.

FIG. 3A depicts an implementation of a method for pausing live video andallowing a user to graphically generate a query by designating aselected region within an image frame. The method of FIG. 3A is carriedout using client device 90 configured to include a telestrator clientapplication and user interface 96. As FIG. 3A shows, the telestratorclient application, when launched, waits for the user to issue a pausecommand (step 202). When the user touches or otherwise activates “pause”button 104, client device 90 transmits a pause command to receiver 36using its infrared port (step 204). Once the pause command is issued,the user provides input by drawing selected region 120 around objects ofinterest within screen display box 102. This input, consisting of linesand curves drawn by the user on a screen display box 102, is telestratordata. The telestrator data can be entered using a stylus on atouch-sensitive display, or using other methods such as a mouse or otherinput device. The objects of interest are objects shown within the imageframe for which the user desires further information. In oneimplementation, the user completes drawing selected region 120 andactivates “done” button 106 to transmit the telestrator data definingselected region 120 to receiver 36.

In the implementation of FIG. 3A, the telestrator data defining selectedregion 120 is displayed on the display device, such as television 38, asit is input by the user. More specifically, the telestrator clientapplication resets a counter (step 206) and, when the counter reaches apredetermined threshold (step 212), client device 90 transmits toreceiver 36 the telestrator data input by the user as it exists when thethreshold is reached (step 218). The counter threshold, in oneimplementation, can be configured such that the telestrator data istransmitted several times every second. For instance, the telestratordata can be transmitted every hundredth of a second, every tenth of asecond, every half-second, once every second, any other time intervalthat is acceptable. When the telestrator data is received by receiver36, the telestrator data is presented on the paused image frame, sowhatever portion of selected region 120 that has been drawn is now shownto the user. This allows the user to monitor his or her progress on thedisplay of television 38 as he or she draws a selected region 120.Therefore, as the user draws a selected region 120 on screen display box102, the lines and curves she is drawing appear on television 38.

When the user is finished drawing selected region 120 in screen displaybox 102, the user activates “done” button 106 (see step 208), causingthe telestrator client application to transmit the telestrator datadefining selected region 120 and a “done” command, including useridentification data (step 214). The telestrator client application thenresets the screen display box 102 (step 220). As FIG. 3A illustrates,the user can cancel the operation at any time by activating “cancel”button 108 (see step 210), thereby causing the telestrator clientapplication to transmit a cancel command to receiver 36 (step 216) andreset screen display box 102 (step 220). In other implementations, userinterface 96 is responsive to a mere tap or scribble on screen 94(rather than the drawing of selected region 120). In theseimplementations, the telestrator data consists of the point where thetap occurred or the lines and curves forming the scribble. Thepositional parameters associated with the tap or scribble can be used toidentify objects of interest, as discussed below.

FIG. 3B sets forth a method implemented by receiver 36 to processcommands received from client device 90 and transmit graphicallygenerated queries to information lookup system 50. In thisimplementation, receiver 36 includes a telestrator receiver applicationthat is invoked upon detection of a command transmitted from clientdevice 90, such as a pause command. As FIG. 3B illustrates, thetelestrator receiver application operates in the background and monitorsfor a pause command (see step 232) while receiver 36 performs its normaloperations of decoding broadcast signals from either DBS system 30 orcable television system 40 and transmitting analog or digital signals totelevision 38 for display. When a pause command is received, thetelestrator receiver application is invoked and receiver 36 pausesnormal transmission of the broadcast stream, thereby displaying a stillimage frame substantially corresponding to the image frame that wasbeing displayed at the moment the pause command was received (step 234).

The telestrator receiver application, in one implementation, stores in amemory of receiver 36 (e.g., a buffer) the image frame data, the timeassociated with the image frame, and the channel to which receiver 36has been tuned, or embedded station identifier and/or frame identifierinformation as obtained from the Vertical Blanking Interval (VBI) data.In another implementation, the station and/or frame information can beembedded in the image data portion of the television signal. For digitalvideo streams, the station and/or frame information can also be embeddedat any suitable location in the stream. The telestrator receiverapplication then monitors for telestrator data and/or other commandsignals transmitted from client device 90.

In one implementation, when the telestrator receiver applicationreceives telestrator data from client device 90 (step 236), thetelestrator receiver application causes receiver 36 to overlay thistelestrator data onto the paused image frame (step 238). Since thetelestrator data from client device 90 generally consists of lines andcurves describing selected region 120, the data overlaid onto the pausedimage frame will also consist of lines and curves corresponding toselected region 120. As discussed above, client device 90 is configuredto periodically transmit updated telestrator data to receiver 36 that isoverlaid onto the paused image frame, thereby allowing the user tomonitor his or her progress on television 38 while drawing selectedregion 120.

If the telestrator receiver application detects a done command (step240), it can retrieve the time and channel/station identifierinformation associated with the paused image frame from the buffer (step246). The telestrator receiver application can transmit a block ofinformation to information lookup system 50. This block of informationcan include the user identification information associated with the donecommand, an image frame identifier (for identifying the specific stillimage frame of the broadcast that was displayed), the telestrator datadefining selected region 120 drawn by the user, as well as the time andchannel/station identifier information (step 248). In oneimplementation, receiver 36 can directly determine, and then transmit,the frame number associated with the paused image frame when framenumbers (and optionally program identifiers) are transmitted in the VBIof the video data stream. The telestrator receiver application thenreturns to the background to monitor for subsequent pause commands,causing receiver 36 to resume normal playback of the broadcasttelevision signal.

Processing of Information Look-Up Requests

Information lookup system 50 includes information database 56 forstoring information related to one or more objects of interestassociated with a television broadcast. Information database 56 can be arelational database including a plurality of related tables. Informationdatabase 56 can maintain, for a given television broadcast or program,an object locator table including fields, as described below, allowingfor identification of one or more objects of interest, if any, includedin telestrator data corresponding to an image frame. More specifically,these objects of interest are generally found in one or more imageframes of a given television broadcast, and can be identifield withinthose image frames by selected region 120, which is defined by thetelestrator data.

To process information lookup requests transmitted from receiver 36,information associated with one or more objects of interest can bestored in relation to image frame and position information. FIG. 4provides one example of an object locator table, according to animplementation of the invention, corresponding to a given televisionprogram. In the implementation of FIG. 4, the object locator tableprovides the positional parameters (x,y) of objects (1 to N) included ina broadcast program relative to a frame or range of frames that make upthe video frame. The frame identifiers can be expressed as time valuesfrom a zero start value (e.g., hh:mm:sec:frame), or as contiguous framenumbers. For each frame or range of frames, the object locator table caninclude x,y positional parameters associated with objects, if any,contained within the image frames. In one implementation, the x,ypositional parameters are x,y pixel values; in another implementation,the x,y parameters are expressed as a composite x-ratio, y-ratiorelative to the aspect ratio of the image. For example, using the upperleft-hand corner of the image as the origin, an x,y coordinate value of(0.25, 0.33) indicates that the x coordinate is ¼ of the image widthhorizontally from the origin, while the y coordinate is ⅓ of the imageheight down from the origin.

For each object, a variety of x,y positional parameters can be used. Forexample, as to a given frame set, the x,y parameters for an object ofinterest can represent a nominal center point of the object within eachframe corresponding to the image frame set. The x,y positionalparameters can also be expressed as a range of x and y coordinatevalues. Furthermore, the nominal center point of a given object may beexpressed as a set of x,y coordinates, all of which must be included inthe telestrator data to be deemed a match. In another implementation,the positional parameters can comprise a set of coordinates defining theoutline of an object in relation to an origin. One of skill in the artwill recognize that myriad combinations are possible.

In one implementation, the producer of a particular broadcast programcan create an object locator table by identifying one or more objects ofinterest for which more information is available. Then, for each object,the producer can populate an object locator table by providing aplurality of frame sets and the corresponding x,y coordinate valuesassociated with the identified objects of interest. In addition, FIG. 5illustrates an object information table that includes computer networkaddresses corresponding to information (e.g., product information,advertisements, and coupons) about the objects of interest.

FIG. 6 illustrates one implementation of a method for processinginformation lookup requests submitted by receiver 36 in response to auser command. As discussed above, an information lookup request, in oneimplementation, comprises a user identifier, a channel or broadcaststation identifier, a time value associated with the pause command, andtelestrator data defining selected region 120 drawn by the user. As FIG.6 shows, when information lookup system 50 receives a lookup request(step 302), it first attempts to identify the program identifiercorresponding to the lookup request (step 304). In one implementation,information lookup system 50 can scan the station and/or channelidentifier and time value against a set of station logs to determine theprogram, ad or other broadcast content corresponding to the lookuprequest. In other implementations, the program identifier may beembedded in the VBI (or the image portion) of the broadcast signal,which is detected by receiver 36 and transmitted, in addition to, or inlieu of, a station identifier. Using the program identifier, informationlookup system 50 then scans information database 56 for an objectlocator table corresponding to the program (step 306).

If an object locator table is found, information lookup system 50determines the frame identifier corresponding to the request (step 308).In one implementation, this can be accomplished by comparing the timevalue associated with the lookup request against the station log tolocate a frame set identifier. In implementations where frameidentification information is embedded in the video signal, such frameidentifiers can be stored by receiver 36 and submitted to informationlookup system 50 as part of the lookup request.

After the appropriate frame or frame set identifier has been determined,information lookup system 50 determines whether selected region 120designated by the user (as defined by the telestrator data) encompassesany of the x,y coordinate values associated with objects for whichinformation is available (steps 310 and 312). If one or more objectidentifiers are found (step 312), information lookup system 50 scans anobject information table (see FIG. 5) to locate information about theobjects (step 314). In one implementation, information lookup system 50then transmits the computer network addresses or the data associatedwith the computer network addresses to the user.

As discussed above, information lookup system 50 maintains a useraccount database 54 including user account identifications andcorresponding email addresses. Using this information, any objectinformation can be packaged in an email and transmitted to the emailaddress associated with the user identification in the lookup request.Of course, one skilled in the art will recognize that a variety ofcontent delivery options are available. For example, information lookupsystem 50 can merely store the content or the results of the lookuprequest in association with the user's account. The lookup results canthen be available to the user during a subsequent login. The lookupresults can also be stored in an on-line synchronization utility to bedownloaded to the user's client device 90 during his or her nextsynchronization operation.

In some implementations, certain features of digital video protocols andtechnologies can be used to facilitate the identification of objectswithin an image frame. The MPEG-4, Level 10 standard (as well ascompeting and/or similar standards) generates an image using multiplesoftware objects. For example, using this standard, a picture may becomposed of a background image object and a few smaller fixed or movingobjects. The audio is delivered in one stream. First the backgroundimage object is established, and this is gradually updated as bandwidthpermits. The center of interest and other objects are drawn and allowedto move in any direction or grow or shrink.

Accordingly, the transmitted data can include instructions regarding howthe objects should change within the image. These types of activitiesuse less bandwidth than a complete re-transmission of these objects orstandard image frames. The background may be a moving window into asingle large image. The most relevant aspect of the MPEG technology tothe present invention is that at the time a user issues a “pause”command, the object visible in the image and designated by the user canbe identified by receiver 36 which is decoding the MPEG signal and isaware of the area of interest designated by the user, whether the objectis encompassed by selection region 120 or “clicked on” by the user usingstylus 92. This object may move across the screen, change over time, ormove due to the perspective of the camera, but it can still beidentified as a specific software object. The user's designated area orstylus click can then be decoded to identify which object was selected.The selected object can then be processed as described above. This or asubsequent generation MPEG Java or other programming capability willallow execution of this routine.

Other video protocols and technologies can also be used in theinvention. For example, the SMPTE (Society of Motion Picture andTelevision Engineers) provides metadata formats that can be deliveredwith the stream of video data to allow for transmission of the show,episode and other metadata. These formats can be extended to supportobject identification for MPEG or equivalent or successor digitalstreams.

Other implementations, in addition to those already disclosed, arepossible. For example, in one implementation, as part of the productionprocess, a predefined set of image frames are created specifically forobject identification purposes. According to one implementation,predefined image frames are provided wherein each predefined image framecorresponds to a plurality of image frames from the underlying video ortelevision program. Accordingly, when a user pauses the transmissionbroadcast for purposes of identifying an object, receiver 36 presentsthe predefined image frame that corresponds to or is associated with theimage frame that was being shown at the time the user paused the videoplayback.

In this implementation, the predefined image frame can contain allobjects in, and can closely resemble, the image frame that correspondsto the time when the user paused the video broadcast. Alternately, thepredefined image frame can look completely dissimilar to the image framethat was being shown when the user paused the video broadcast. Forexample, if the image frame that was being displayed when the usertransmits the pause command has three objects of interest, thepredefined image frame that is now shown by receiver 36 can simplycontain three objects of interest. This makes identifying the objectthat the user is interested in an easier task. This implementationgreatly reduces the number of frames for which object locations must beidentified. In one implementation, the set of predefined image framescan be transmitted to receiver 36 and stored in a buffer or on a harddisk for access as required. In another implementation, receiver 36transmits requests for a predefined image frame from a predefined setstored at a remote location, such as DBS system 30 or cable televisionsystem 40.

In another implementation, receiver 36 transmits a presentation of theimage frame associated with the pause command to client device 90 fordisplay on screen 94 to allow a user to view the objects of interest inthe screen display box 102 and more closely define selected region 120.In such an implementation, receiver 36 not pause playback of the videobroadcast.

FIG. 7 illustrates an implementation where the methods of the inventioncan be carried out without the use of receiver 36. In thisimplementation, the commands for a given image frame can be transmittedbetween client device 90 and information lookup system 50 over awireless communications medium. In one implementation, the functionalityof information lookup system 50 is incorporated into a televisionbroadcast station 80. Television broadcast station 80 and informationlookup system 50 can communicate data such as station logs and imageframe files over computer network 20.

In this implementation, when a user sees an object during a televisionbroadcast about which he desires more information, the user can activatea command on client device 90 that is communicated via wireless network21 and WAP gateway 22 to information lookup system 50. This command issimilar to the pause command discussed above and identifies the channelto which television 38 is tuned and optionally includes a time stamp.Information lookup system 50, having information regarding the tunedchannel and the time (either by when the command was received or theoptional time stamp), can then determine the image frame number or rangeof frame numbers that is likely to have been displayed at the time theuser issued the pause command. Information lookup system 50 can thentransmit to client device 90 one or more image frames from its filesthat are similar to the image frame displayed about the time that the“pause” command was transmitted, along with the positional parametersassociated with one or more objects of interest in the image.

In one implementation, the image transmitted to client device 90 frominformation lookup system 50 can be a composite of two or more scenes inthe event that the scenes change frequently in the segment of of thetelevision program that the “pause” command was issued; this can addressimplementations of the system where resolution of the frame number isinhibited by delays associated with the system and/or the user inactivating or initiating a pause command. In one implementation, thepositional parameters include one corner x,y and a distance to oppositecorner Δx,Δy. In one embodiment, multiple boxes (as defined bycorner-delta values) may be required to form the item.

Upon receipt of the image frame data from information lookup system 50,client device 90 displays the image frame on the user interface 96,allowing the user to define a selected region 120 including one or moreobjects of interest. In one implementation, the client applicationinstalled on client device 90 reviews the seleced region 120 (or tappoint or scribble) drawn by the user against the positional parametersassociated with the image frame to find at least a few points or pixelsinside one of the boxes defining the available objects. If any objectsare found, the object identifiers are transmitted to information lookupsystem 50 over wireless network 21 and WAP gateway 22, and informationlookup system 50 can return information relating to the one or moreobjects of interest.

The invention has been described with reference to specificimplementations. Other implementations of the invention will be apparentto those of ordinary skill in the art. For example, the functionality ofinformation lookup system 50 may be integrated into DBS system 30 orcable television system 40. It is, therefore, intended that the scope ofthe invention not be limited to the implementations described above.

1. A method for requesting information about an object displayed in abroadcast stream, the method comprising: receiving at a receiver abroadcast stream; transmitting from the receiver the broadcast stream toa display device; receiving at the receiver a command from a clientdevice to pause the broadcast stream displayed on the display device;pausing the broadcast stream displayed on the display device responsiveto receiving the command; transmitting from the receiver a still imageframe from the paused broadcast stream to the display device, the stillimage frame comprising a frame of the paused broadcast stream that wasdisplayed on the display device when the command was received from theclient device; receiving at the receiver telestrator data designating atleast a portion of the transmitted still image frame that designates alocation of an object displayed in the portion of the transmitted stillimage frame; generating at the receiver a query for information aboutthe object displayed in the portion of the transmitted still imageframe, the query comprising the telestrator data that designates thelocation of the object in the portion of the transmitted still imageframe; and transmitting from the receiver the query to a remoteinformation system.
 2. The method of claim 1, wherein the telestratordata is displayed on the display device.
 3. The method of claim 2,wherein the telestrator data is overlaid onto the still image framedisplayed on the display device.
 4. The method of claim 1, wherein thebroadcast stream comprises a satellite broadcast stream or a cablebroadcast stream.
 5. The method of claim 1, wherein the broadcast streamcomprises a previously recorded broadcast stream stored on a storagedevice.
 6. The method of claim 1, wherein the still image framecomprises a predefined image frame that corresponds to the image frameof the broadcast stream that was displayed on display device when thecommand was received from the client device.
 7. The method of claim 1,wherein the telestrator data comprises lines and/or curves enclosing theobject of interest within the still image frame.
 8. The method of claim1, wherein the telestrator data comprises one or more pixels placeddirectly atop the object of interest within the still image frame. 9.The method of claim 1, wherein the telestrator data comprises a scribbleplaced directly atop the object of interest within the still imageframe.
 10. The method of claim 1, wherein the object of interest is anobject shown in the still image frame for which the user desires furtherinformation.
 11. The method of claim 1, wherein transmitting from thereceiver the query to a remote information system is carried out over atelephone network.
 12. The method of claim 1, wherein transmitting fromthe receiver the query to a remote information system is carried outover a computer network.
 13. The method of claim 12, wherein thecomputer network comprises the Internet.
 14. The method of claim 1,wherein transmitting from the receiver the query to a remote informationsystem is carried out over a wireless network.
 15. The method of claim1, wherein the commands and the data are received in the form ofwireless signals.
 16. The method of claim 15, wherein the wirelesssignals comprise infrared signals.
 17. The method of claim 15, whereinthe wireless signals comprise Bluetooth signals.
 18. The method of claim15, wherein the wireless signals comprise 802.11 signals.
 19. The methodof claim 1, wherein the data identifying the still image frame comprisesa frame number.
 20. The method of claim 1, wherein the data identifyingthe still image frame comprises a time value.
 21. The method of claim 1,wherein the data identifying the still image frame is found in thevertical blanking interval.
 22. The method of claim 1, wherein the dataidentifying the still image frame is found in the broadcast stream. 23.The method of claim 1, wherein the data identifying the still imageframe includes a program identifier.
 24. The method of claim 1, whereinthe display device comprises a television.
 25. An apparatus forrequesting information about an object displayed in a broadcast stream,the apparatus comprising: a processor; a wireless communicationreceiver; a computer-readable storage medium comprising computerexecutable instructions when executed by the processor: receive abroadcast stream by the wireless communications receiver; transmit thebroadcast stream to a display device; receive a command from a clientdevice to pause the broadcast displayed on the display device; pause thebroadcast stream displayed on the display device responsive to receivingthe command; transmit a still image frame from the paused broadcaststream to the display device, the still image frame comprising a frameof the paused broadcast stream that was displayed on the display devicewhen the command was received from the client device; receivetelestrator data designating at least a portion of the transmitted stillimage frame that designates a location of an object displayed in theportion of the transmitted still image frame; generate a query forinformation about the object displayed in the portion of the transmittedstill image frame, the query comprising the telestrator data thatdesignates the location of the object in the portion of the transmittedstill image frame; and transmit the query to a remote informationsystem.
 26. The apparatus of claim 25, wherein the wirelesscommunications system comprises an infrared communications system. 27.The apparatus of claim 25, wherein the wireless communications systemcomprises a Bluetooth communications system.
 28. The apparatus of claim25, wherein the wireless communications system comprises an 802.11communications system.
 29. The apparatus of claim 25, wherein thewherein the computer executable instructions when executed by theprocessor display the telestrator data on the display device.
 30. Theapparatus of claim 29, wherein the telestrator data is overlaid onto thestill image frame displayed on the display device.
 31. The apparatus ofclaim 25, wherein the still image frame comprises a predefined imageframe that corresponds to the image frame of the broadcast stream thatwas displayed on display device when the command was received from theclient device.
 32. The apparatus of claim 25, wherein the commands andthe data are received in the form of wireless signals.
 33. The apparatusof claim 32, wherein the wireless signals comprise infrared signals,Bluetooth signals, or 802.11 signals.
 34. The apparatus of claim 25,wherein the data identifying the still image frame includes a programidentifier.
 35. The apparatus of claim 25, wherein transmitting thequery to a remote information system is carried out over a telephonenetwork, a computer network, or a wireless network.
 36. A computerprogram product comprising a non-transitory computer readable storagemedium for requesting information about an object displayed in abroadcast stream, the non-transitory computer readable storage mediumstoring executable instructions that cause a processor to: receiving ata receiver a broadcast stream; transmit from the receiver the broadcaststream to a display device; receive at the receiver a command from aclient device to pause the broadcast stream displayed on the displaydevice pausing the broadcast stream displayed on the display deviceresponsive to receiving the command; transmit from the receiver a stillimage frame from the paused broadcast stream to the display device, thestill image frame comprising a frame of the paused broadcast stream thatwas displayed on the display device when the command was received fromthe client device; receive at the receiver telestrator data designatingat least a portion of the transmitted still image frame that designatesa location of an object displayed in the portion of the transmittedstill image frame; generate at the receiver a query for informationabout the object displayed in the portion of the transmitted still imageframe, the query comprising the telestrator data that designates thelocation of the object in the portion of the transmitted still imageframe; and transmit from the receiver the query to a remote informationsystem.